  Recent breakthroughs in the field of language-guided image generation have
yielded impressive achievements, enabling the creation of high-quality and
diverse images based on user instructions.Although the synthesis performance is
fascinating, one significant limitation of current image generation models is
their insufficient ability to generate text coherently within images,
particularly for complex glyph structures like Chinese characters. To address
this problem, we introduce GlyphDraw, a general learning framework aiming to
endow image generation models with the capacity to generate images coherently
embedded with text for any specific language.We first sophisticatedly design
the image-text dataset's construction strategy, then build our model
specifically on a diffusion-based image generator and carefully modify the
network structure to allow the model to learn drawing language characters with
the help of glyph and position information.Furthermore, we maintain the model's
open-domain image synthesis capability by preventing catastrophic forgetting by
using parameter-efficient fine-tuning techniques.Extensive qualitative and
quantitative experiments demonstrate that our method not only produces accurate
language characters as in prompts, but also seamlessly blends the generated
text into the background.Please refer to our
\href{https://1073521013.github.io/glyph-draw.github.io/}{project page}.
\end{abstract}

Computer Vision and Pattern Recognition