TikZero: Zero-Shot Text-Guided Graphics Program Synthesis
With the rise of generative AI, synthesizing figures from text captions
becomes a compelling application. However, achieving high geometric precision
and editability requires representing figures as graphics programs in languages
like TikZ, and aligned training data (i.e., graphics programs with captions)
remains scarce. Meanwhile, large amounts of unaligned graphics programs and
captioned raster images are more readily available. We reconcile these
disparate data sources by presenting TikZero, which decouples graphics program
generation from text understanding by using image representations as an
intermediary bridge. It enables independent training on graphics programs and
captioned images and allows for zero-shot text-guided graphics program
synthesis during inference. We show that our method substantially outperforms
baselines that can only operate with caption-aligned graphics programs.
Furthermore, when leveraging caption-aligned graphics programs as a
complementary training signal, TikZero matches or exceeds the performance of
much larger models, including commercial systems like GPT-4o. Our code,
datasets, and select models are publicly available.
67d7e187ee9f6c22e2b1fa0a
Jonas Belouadi
67d7e187ee9f6c22e2b1fa0b
Eddy Ilg
67d7e187ee9f6c22e2b1fa0c
Margret Keuper
67d7e187ee9f6c22e2b1fa0d
Hideki Tanaka
67d7e187ee9f6c22e2b1fa0e
Masao Utiyama
67d7e187ee9f6c22e2b1fa0f
Raj Dabre
67d7e187ee9f6c22e2b1fa10
Steffen Eger
67d7e187ee9f6c22e2b1fa11
Simone Paolo Ponzetto
TikZero: Zero-Shot Text-Guided Graphics Program Synthesis
With the rise of generative AI, synthesizing figures from text captions
becomes a compelling application. However, achieving high geometric precision
and editability requires representing figures as graphics programs in languages
like TikZ, and aligned training data (i.e., graphics programs with captions)
remains scarce. Meanwhile, large amounts of unaligned graphics programs and
captioned raster images are more readily available. We reconcile these
disparate data sources by presenting TikZero, which decouples graphics program
generation from text understanding by using image representations as an
intermediary bridge. It enables independent training on graphics programs and
captioned images and allows for zero-shot text-guided graphics program
synthesis during inference. We show that our method substantially outperforms
baselines that can only operate with caption-aligned graphics programs.
Furthermore, when leveraging caption-aligned graphics programs as a
complementary training signal, TikZero matches or exceeds the performance of
much larger models, including commercial systems like GPT-4o. Our code,
datasets, and select models are publicly available.
TikZero uses image representations as a bridge to synthesize graphics programs from text captions, outperforming baselines and matching larger models in zero-shot and supervised settings.
generative AI TikZ graphics programs captioned raster images image representations text-guided graphics synthesis zero-shot synthesis