title: Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation
summary: Significant progress has recently been made in creative applications of large
  pre-trained models for downstream tasks in 3D vision, such as text-to-shape
  generation. This motivates our investigation of how these pre-trained models
  can be used effectively to generate 3D shapes from sketches, which has largely
  remained an open challenge due to the limited sketch-shape paired datasets and
  the varying level of abstraction in the sketches. We discover that conditioning
  a 3D generative model on the features (obtained from a frozen large pre-trained
  vision model) of synthetic renderings during training enables us to effectively
  generate 3D shapes from sketches at inference time. This suggests that the
  large pre-trained vision model features carry semantic signals that are
  resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
  generalizing to sketches at inference time. We conduct a comprehensive set of
  experiments investigating different design factors and demonstrate the
  effectiveness of our straightforward approach for generation of multiple 3D
  shapes per each input sketch regardless of their level of abstraction without
  requiring any paired datasets during training.
paper_authors._id: 64acc2decf90fe275545b3eb
paper_authors.name: Aditya Sanghi
paper_authors._id: 64acc2decf90fe275545b3ec
paper_authors.name: Pradeep Kumar Jayaraman
paper_authors._id: 64acc2decf90fe275545b3ed
paper_authors.name: Arianna Rampini
paper_authors._id: 64acc2decf90fe275545b3ee
paper_authors.name: Joseph Lambourne
paper_authors._id: 64acc2decf90fe275545b3ef
paper_authors.name: Hooman Shayani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T07:29:38.014Z
paper_authors._id: 627a75d3adee2d5829796aeb
paper_authors.avatarUrl: /avatars/143b2717b7af7f1d0334d079b6d65d09.svg
paper_authors.fullname: Shayani
paper_authors.type: user
paper_authors.user: Hooman
paper_authors._id: 64acc2decf90fe275545b3f0
paper_authors.name: Evan Atherton
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:49:01.202Z
paper_authors._id: 63575f59e75acf88c90588ee
paper_authors.avatarUrl: /avatars/056fbd40039ede19de400defd18eff11.svg
paper_authors.fullname: Evan Atherton
paper_authors.type: user
paper_authors.user: boshwan527
paper_authors._id: 64acc2decf90fe275545b3f1
paper_authors.name: Saeid Asgari Taghanaki
paper_title: Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation
paper_summary: Significant progress has recently been made in creative applications of large
  pre-trained models for downstream tasks in 3D vision, such as text-to-shape
  generation. This motivates our investigation of how these pre-trained models
  can be used effectively to generate 3D shapes from sketches, which has largely
  remained an open challenge due to the limited sketch-shape paired datasets and
  the varying level of abstraction in the sketches. We discover that conditioning
  a 3D generative model on the features (obtained from a frozen large pre-trained
  vision model) of synthetic renderings during training enables us to effectively
  generate 3D shapes from sketches at inference time. This suggests that the
  large pre-trained vision model features carry semantic signals that are
  resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
  generalizing to sketches at inference time. We conduct a comprehensive set of
  experiments investigating different design factors and demonstrate the
  effectiveness of our straightforward approach for generation of multiple 3D
  shapes per each input sketch regardless of their level of abstraction without
  requiring any paired datasets during training.
paper_ai_keywords: 3D vision text-to-shape generation pre-trained models sketch-shape paired datasets 3D generative model frozen large pre-trained vision model synthetic renderings semantic signals domain shifts RGB renderings abstraction paired datasets