title: Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
summary: Visual text rendering poses a fundamental challenge for contemporary
  text-to-image generation models, with the core problem lying in text encoder
  deficiencies. To achieve accurate text rendering, we identify two crucial
  requirements for text encoders: character awareness and alignment with glyphs.
  Our solution involves crafting a series of customized text encoder, Glyph-ByT5,
  by fine-tuning the character-aware ByT5 encoder using a meticulously curated
  paired glyph-text dataset. We present an effective method for integrating
  Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for
  design image generation. This significantly enhances text rendering accuracy,
  improving it from less than 20% to nearly 90% on our design image
  benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph
  rendering, achieving high spelling accuracy for tens to hundreds of characters
  with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with
  a small set of high-quality, photorealistic images featuring visual text, we
  showcase a substantial improvement in scene text rendering capabilities in
  open-domain real images. These compelling outcomes aim to encourage further
  exploration in designing customized text encoders for diverse and challenging
  tasks.
paper_authors._id: 65f3b4173a0b542b87c5bd43
paper_authors.name: Zeyu Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T14:36:45.607Z
paper_authors._id: 64f2a14396a5ffb820ab39a8
paper_authors.avatarUrl: /avatars/3d7c3be17c53eb52bd8a75d3c51e6891.svg
paper_authors.fullname: Zeyu Liu
paper_authors.type: user
paper_authors.user: shuidi0020
paper_authors._id: 65f3b4173a0b542b87c5bd44
paper_authors.name: Weicong Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:30:13.682Z
paper_authors._id: 6437ac5dd38ce48bdd4a4e00
paper_authors.avatarUrl: /avatars/5d183d9c55043ccfc33cc58c86243182.svg
paper_authors.fullname: Weicong Liang
paper_authors.type: user
paper_authors.user: kxqt
paper_authors._id: 65f3b4173a0b542b87c5bd45
paper_authors.name: Zhanhao Liang
paper_authors._id: 65f3b4173a0b542b87c5bd46
paper_authors.name: Chong Luo
paper_authors._id: 65f3b4173a0b542b87c5bd47
paper_authors.name: Ji Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T18:36:59.879Z
paper_authors._id: 6473768cbf9b32c6bbc9caa6
paper_authors.avatarUrl: /avatars/7df22e0f33aac855a5d2b0e9286c7ad5.svg
paper_authors.fullname: aaa
paper_authors.type: user
paper_authors.user: jiangsuliji
paper_authors._id: 65f3b4173a0b542b87c5bd48
paper_authors.name: Gao Huang
paper_authors._id: 65f3b4173a0b542b87c5bd49
paper_authors.name: Yuhui Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:30:52.020Z
paper_authors._id: 646f69a6041e48e1c4728de3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f69a6041e48e1c4728de3/U5OaW6PgsXTXnfG03xs9Q.png
paper_authors.fullname: GlyphByT5
paper_authors.type: user
paper_authors.user: GlyphByT5
paper_title: Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
paper_summary: Visual text rendering poses a fundamental challenge for contemporary
  text-to-image generation models, with the core problem lying in text encoder
  deficiencies. To achieve accurate text rendering, we identify two crucial
  requirements for text encoders: character awareness and alignment with glyphs.
  Our solution involves crafting a series of customized text encoder, Glyph-ByT5,
  by fine-tuning the character-aware ByT5 encoder using a meticulously curated
  paired glyph-text dataset. We present an effective method for integrating
  Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for
  design image generation. This significantly enhances text rendering accuracy,
  improving it from less than 20% to nearly 90% on our design image
  benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph
  rendering, achieving high spelling accuracy for tens to hundreds of characters
  with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with
  a small set of high-quality, photorealistic images featuring visual text, we
  showcase a substantial improvement in scene text rendering capabilities in
  open-domain real images. These compelling outcomes aim to encourage further
  exploration in designing customized text encoders for diverse and challenging
  tasks.
paper_ai_summary: Customizing the ByT5 text encoder with Glyph-ByT5 and integrating it with SDXL significantly improves text rendering accuracy and paragraph rendering in design and real images.
paper_ai_keywords: text encoder Glyph-ByT5 ByT5 fine-tuning paired glyph-text dataset SDXL Glyph-SDXL text rendering accuracy automated multi-line layouts scene text rendering