Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches
Itsumi Saito Haruto Yoshida Keisuke Sakaguchi
We address the challenge of automatically generating high-quality vector diagrams from hand-drawn sketches. 
Vector diagrams are essential for communicating complex ideas across various fields, offering flexibility and scalability. 
While recent research has progressed in generating diagrams from text descriptions, converting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets. 
To address this gap, we introduce SketikZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams.
Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SketikZ as a key benchmark for future research in sketch-to-diagram conversion.
Along with SketikZ, we present ImgTikZ, an image-to-TikZ model that integrates a 6.7B parameter code-specialized open-source large language model (LLM) with a pre-trained vision encoder. 
Despite its relatively compact size, ImgTikZ performs comparably to GPT-4o.
This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy.
Our findings open promising directions for future research in sketch-to-diagram conversion and broader image-to-code generation tasks. SketikZ is publicly available.
multimodal, large language model, diagram, vector graphics
We address the challenge of automatically converting hand-drawn sketches into vector diagrams. We present SkeTikZ, a dataset of 3,231 sketch-diagram-code triplets, and ImgTikZ, a vision-language model that performs the sketch-to-diagram conversion.