TextDiffuser: Diffusion Models as Text Painters
Diffusion models have gained increasing attention for their impressive
generation abilities but currently struggle with rendering accurate and
coherent text. To address this issue, we introduce TextDiffuser,
focusing on generating images with visually appealing text that is coherent
with backgrounds. TextDiffuser consists of two stages: first, a Transformer
model generates the layout of keywords extracted from text prompts, and then
diffusion models generate images conditioned on the text prompt and the
generated layout. Additionally, we contribute the first large-scale text images
dataset with OCR annotations, MARIO-10M, containing 10 million
image-text pairs with text recognition, detection, and character-level
segmentation annotations. We further collect the MARIO-Eval benchmark
to serve as a comprehensive tool for evaluating text rendering quality. Through
experiments and user studies, we show that TextDiffuser is flexible and
controllable to create high-quality text images using text prompts alone or
together with text template images, and conduct text inpainting to reconstruct
incomplete images with text. The code, model, and dataset will be available at
https://aka.ms/textdiffuser.
6466db2856372fc9e3e8d4dc
Jingye Chen
claimed_verified
2023-05-19T14:33:52.984Z
63355049ce9a15ea6c4aa8cc
https://cdn-avatars.huggingface.co/v1/production/uploads/63355049ce9a15ea6c4aa8cc/aD4NI5d_ozAuQVv5nsNF4.jpeg
JingyeChen
user
JingyeChen
6466db2856372fc9e3e8d4dd
Yupan Huang
claimed_verified
2023-05-19T13:18:21.657Z
625d7fc87341c641426c7665
/avatars/3083ebe0343a03332b15f0e7f76ba1d7.svg
Yupan Huang
user
HYPJUDY
6466db2856372fc9e3e8d4de
Tengchao Lv
6466db2856372fc9e3e8d4df
Lei Cui
claimed_verified
2023-05-19T13:18:17.753Z
5f5a4fbe10b2753d9000c7fc
/avatars/0763b25c13733ef4eb757862ba6d603e.svg
Lei Cui
user
wolfshow
6466db2856372fc9e3e8d4e0
Qifeng Chen
claimed_verified
2023-05-19T19:04:26.867Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
6466db2856372fc9e3e8d4e1
Furu Wei
TextDiffuser: Diffusion Models as Text Painters
Diffusion models have gained increasing attention for their impressive
generation abilities but currently struggle with rendering accurate and
coherent text. To address this issue, we introduce TextDiffuser,
focusing on generating images with visually appealing text that is coherent
with backgrounds. TextDiffuser consists of two stages: first, a Transformer
model generates the layout of keywords extracted from text prompts, and then
diffusion models generate images conditioned on the text prompt and the
generated layout. Additionally, we contribute the first large-scale text images
dataset with OCR annotations, MARIO-10M, containing 10 million
image-text pairs with text recognition, detection, and character-level
segmentation annotations. We further collect the MARIO-Eval benchmark
to serve as a comprehensive tool for evaluating text rendering quality. Through
experiments and user studies, we show that TextDiffuser is flexible and
controllable to create high-quality text images using text prompts alone or
together with text template images, and conduct text inpainting to reconstruct
incomplete images with text. The code, model, and dataset will be available at
https://aka.ms/textdiffuser.
TextDiffuser, a two-stage model combining Transformer and diffusion models, generates visually appealing and coherent text images using text prompts and layouts, leveraging the MARIO-10M dataset for training and MARIO-Eval for evaluation.
TextDiffuser Transformer model diffusion models text prompts visual coherence OCR annotations MARIO-10M MARIO-Eval text rendering quality text inpainting