title: TextDiffuser: Diffusion Models as Text Painters
summary: Diffusion models have gained increasing attention for their impressive
  generation abilities but currently struggle with rendering accurate and
  coherent text. To address this issue, we introduce TextDiffuser,
  focusing on generating images with visually appealing text that is coherent
  with backgrounds. TextDiffuser consists of two stages: first, a Transformer
  model generates the layout of keywords extracted from text prompts, and then
  diffusion models generate images conditioned on the text prompt and the
  generated layout. Additionally, we contribute the first large-scale text images
  dataset with OCR annotations, MARIO-10M, containing 10 million
  image-text pairs with text recognition, detection, and character-level
  segmentation annotations. We further collect the MARIO-Eval benchmark
  to serve as a comprehensive tool for evaluating text rendering quality. Through
  experiments and user studies, we show that TextDiffuser is flexible and
  controllable to create high-quality text images using text prompts alone or
  together with text template images, and conduct text inpainting to reconstruct
  incomplete images with text. The code, model, and dataset will be available at
  https://aka.ms/textdiffuser.
paper_authors._id: 6466db2856372fc9e3e8d4dc
paper_authors.name: Jingye Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T14:33:52.984Z
paper_authors._id: 63355049ce9a15ea6c4aa8cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63355049ce9a15ea6c4aa8cc/aD4NI5d_ozAuQVv5nsNF4.jpeg
paper_authors.fullname: JingyeChen
paper_authors.type: user
paper_authors.user: JingyeChen
paper_authors._id: 6466db2856372fc9e3e8d4dd
paper_authors.name: Yupan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T13:18:21.657Z
paper_authors._id: 625d7fc87341c641426c7665
paper_authors.avatarUrl: /avatars/3083ebe0343a03332b15f0e7f76ba1d7.svg
paper_authors.fullname: Yupan Huang
paper_authors.type: user
paper_authors.user: HYPJUDY
paper_authors._id: 6466db2856372fc9e3e8d4de
paper_authors.name: Tengchao Lv
paper_authors._id: 6466db2856372fc9e3e8d4df
paper_authors.name: Lei Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T13:18:17.753Z
paper_authors._id: 5f5a4fbe10b2753d9000c7fc
paper_authors.avatarUrl: /avatars/0763b25c13733ef4eb757862ba6d603e.svg
paper_authors.fullname: Lei Cui
paper_authors.type: user
paper_authors.user: wolfshow
paper_authors._id: 6466db2856372fc9e3e8d4e0
paper_authors.name: Qifeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:26.867Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 6466db2856372fc9e3e8d4e1
paper_authors.name: Furu Wei
paper_title: TextDiffuser: Diffusion Models as Text Painters
paper_summary: Diffusion models have gained increasing attention for their impressive
  generation abilities but currently struggle with rendering accurate and
  coherent text. To address this issue, we introduce TextDiffuser,
  focusing on generating images with visually appealing text that is coherent
  with backgrounds. TextDiffuser consists of two stages: first, a Transformer
  model generates the layout of keywords extracted from text prompts, and then
  diffusion models generate images conditioned on the text prompt and the
  generated layout. Additionally, we contribute the first large-scale text images
  dataset with OCR annotations, MARIO-10M, containing 10 million
  image-text pairs with text recognition, detection, and character-level
  segmentation annotations. We further collect the MARIO-Eval benchmark
  to serve as a comprehensive tool for evaluating text rendering quality. Through
  experiments and user studies, we show that TextDiffuser is flexible and
  controllable to create high-quality text images using text prompts alone or
  together with text template images, and conduct text inpainting to reconstruct
  incomplete images with text. The code, model, and dataset will be available at
  https://aka.ms/textdiffuser.
paper_ai_keywords: diffusion models Transformation TextDiffuser text prompts image generation layout keywords MARIO-10M MARIO-Eval text rendering text images OCR annotations text recognition text detection character-level segmentation text inpainting user studies high-quality text images text template images