title: Text-to-CAD Generation Through Infusing Visual Feedback in Large
    Language Models
summary: Creating Computer-Aided Design (CAD) models requires significant expertise
  and effort. Text-to-CAD, which converts textual descriptions into CAD
  parametric sequences, is crucial in streamlining this process. Recent studies
  have utilized ground-truth parametric sequences, known as sequential signals,
  as supervision to achieve this goal. However, CAD models are inherently
  multimodal, comprising parametric sequences and corresponding rendered visual
  objects. Besides,the rendering process from parametric sequences to visual
  objects is many-to-one. Therefore, both sequential and visual signals are
  critical for effective training. In this work, we introduce CADFusion, a
  framework that uses Large Language Models (LLMs) as the backbone and alternates
  between two training stages: the sequential learning (SL) stage and the visual
  feedback (VF) stage. In the SL stage, we train LLMs using ground-truth
  parametric sequences, enabling the generation of logically coherent parametric
  sequences. In the VF stage, we reward parametric sequences that render into
  visually preferred objects and penalize those that do not, allowing LLMs to
  learn how rendered visual objects are perceived and evaluated. These two stages
  alternate throughout the training, ensuring balanced learning and preserving
  benefits of both signals. Experiments demonstrate that CADFusion significantly
  improves performance, both qualitatively and quantitatively.
paper_authors._id: 67a33e60b793ca5296f2a6d1
paper_authors.name: Ruiyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T13:37:40.966Z
paper_authors._id: 67a30a243d5a32b36c7d7d0b
paper_authors.avatarUrl: /avatars/4aa43f0d79797be1063f246d63638a85.svg
paper_authors.fullname: Ruiyu Wang
paper_authors.type: user
paper_authors.user: rywang37
paper_authors._id: 67a33e60b793ca5296f2a6d2
paper_authors.name: Yu Yuan
paper_authors._id: 67a33e60b793ca5296f2a6d3
paper_authors.name: Shizhao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:15:14.093Z
paper_authors._id: 63eb00a191a1b8ec4fbba2a9
paper_authors.avatarUrl: /avatars/0cc7cf9b6d05337603f700e0d592edf5.svg
paper_authors.fullname: ShizhaoSun
paper_authors.type: user
paper_authors.user: ShizhaoSun
paper_authors._id: 67a33e60b793ca5296f2a6d4
paper_authors.name: Jiang Bian
paper_title: Text-to-CAD Generation Through Infusing Visual Feedback in Large
    Language Models
paper_summary: Creating Computer-Aided Design (CAD) models requires significant expertise
  and effort. Text-to-CAD, which converts textual descriptions into CAD
  parametric sequences, is crucial in streamlining this process. Recent studies
  have utilized ground-truth parametric sequences, known as sequential signals,
  as supervision to achieve this goal. However, CAD models are inherently
  multimodal, comprising parametric sequences and corresponding rendered visual
  objects. Besides,the rendering process from parametric sequences to visual
  objects is many-to-one. Therefore, both sequential and visual signals are
  critical for effective training. In this work, we introduce CADFusion, a
  framework that uses Large Language Models (LLMs) as the backbone and alternates
  between two training stages: the sequential learning (SL) stage and the visual
  feedback (VF) stage. In the SL stage, we train LLMs using ground-truth
  parametric sequences, enabling the generation of logically coherent parametric
  sequences. In the VF stage, we reward parametric sequences that render into
  visually preferred objects and penalize those that do not, allowing LLMs to
  learn how rendered visual objects are perceived and evaluated. These two stages
  alternate throughout the training, ensuring balanced learning and preserving
  benefits of both signals. Experiments demonstrate that CADFusion significantly
  improves performance, both qualitatively and quantitatively.
paper_ai_summary: CADFusion uses alternating sequential and visual training stages with LLMs to enhance the generation of parametric sequences for CAD models from textual descriptions.
paper_ai_keywords: Large Language Models (LLMs) sequential learning visual feedback CADFusion parametric sequences CAD models text-to-CAD