Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation
Significant progress has recently been made in creative applications of large
pre-trained models for downstream tasks in 3D vision, such as text-to-shape
generation. This motivates our investigation of how these pre-trained models
can be used effectively to generate 3D shapes from sketches, which has largely
remained an open challenge due to the limited sketch-shape paired datasets and
the varying level of abstraction in the sketches. We discover that conditioning
a 3D generative model on the features (obtained from a frozen large pre-trained
vision model) of synthetic renderings during training enables us to effectively
generate 3D shapes from sketches at inference time. This suggests that the
large pre-trained vision model features carry semantic signals that are
resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
generalizing to sketches at inference time. We conduct a comprehensive set of
experiments investigating different design factors and demonstrate the
effectiveness of our straightforward approach for generation of multiple 3D
shapes per each input sketch regardless of their level of abstraction without
requiring any paired datasets during training.
64acc2decf90fe275545b3eb
Aditya Sanghi
64acc2decf90fe275545b3ec
Pradeep Kumar Jayaraman
64acc2decf90fe275545b3ed
Arianna Rampini
64acc2decf90fe275545b3ee
Joseph Lambourne
64acc2decf90fe275545b3ef
Hooman Shayani
claimed_verified
2023-07-12T07:29:38.014Z
627a75d3adee2d5829796aeb
/avatars/143b2717b7af7f1d0334d079b6d65d09.svg
Shayani
user
Hooman
64acc2decf90fe275545b3f0
Evan Atherton
claimed_verified
2023-07-17T06:49:01.202Z
63575f59e75acf88c90588ee
/avatars/056fbd40039ede19de400defd18eff11.svg
Evan Atherton
user
boshwan527
64acc2decf90fe275545b3f1
Saeid Asgari Taghanaki
Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation
Significant progress has recently been made in creative applications of large
pre-trained models for downstream tasks in 3D vision, such as text-to-shape
generation. This motivates our investigation of how these pre-trained models
can be used effectively to generate 3D shapes from sketches, which has largely
remained an open challenge due to the limited sketch-shape paired datasets and
the varying level of abstraction in the sketches. We discover that conditioning
a 3D generative model on the features (obtained from a frozen large pre-trained
vision model) of synthetic renderings during training enables us to effectively
generate 3D shapes from sketches at inference time. This suggests that the
large pre-trained vision model features carry semantic signals that are
resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
generalizing to sketches at inference time. We conduct a comprehensive set of
experiments investigating different design factors and demonstrate the
effectiveness of our straightforward approach for generation of multiple 3D
shapes per each input sketch regardless of their level of abstraction without
requiring any paired datasets during training.
A pre-trained vision model's features enable generation of 3D shapes from sketches without paired datasets by leveraging synthetic renderings during training.
pre-trained models text-to-shape generation 3D generative model synthetic renderings semantic signals domain shifts RGB renderings